{
 "cells": [
  {
   "cell_type": "markdown",
   "source": [
    "学习目标\n",
    "- 了解Pandas的几种文件读取存储操作\n",
    "- 应用CSV方式、HDF方式和json方式实现文件的读取和存储"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "145209eb3fb49077"
  },
  {
   "cell_type": "markdown",
   "source": [
    "我们的数据大部分存在于文件当中，所以pandas会支持复杂的IO操作，pandas的API支持众多的文件格式，如CSV、SQL、XLS、JSON、HDF5。\n",
    "> 注：最常用的HDF5和CSV文件\n",
    "\n",
    "# 1 CSV\n",
    "## 1.1 read_csv\n",
    "pandas.read_csv(filepath_or_buffer, sep =',', usecols )\n",
    "参数：\n",
    "- filepath_or_buffer:文件路径\n",
    "- sep :分隔符，默认用\",\"隔开\n",
    "- usecols:指定读取的列名，列表形式"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "849c4d1f620e0405"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [],
   "metadata": {
    "collapsed": false
   },
   "id": "a3eda4408ea6f55a"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "             open  close\n2018-02-27  23.53  24.16\n2018-02-26  22.80  23.53\n2018-02-23  22.88  22.82\n2018-02-22  22.25  22.28\n2018-02-14  21.49  21.92\n...           ...    ...\n2015-03-06  13.17  14.28\n2015-03-05  12.88  13.16\n2015-03-04  12.80  12.90\n2015-03-03  12.52  12.70\n2015-03-02  12.25  12.52\n\n[643 rows x 2 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>open</th>\n      <th>close</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2018-02-27</th>\n      <td>23.53</td>\n      <td>24.16</td>\n    </tr>\n    <tr>\n      <th>2018-02-26</th>\n      <td>22.80</td>\n      <td>23.53</td>\n    </tr>\n    <tr>\n      <th>2018-02-23</th>\n      <td>22.88</td>\n      <td>22.82</td>\n    </tr>\n    <tr>\n      <th>2018-02-22</th>\n      <td>22.25</td>\n      <td>22.28</td>\n    </tr>\n    <tr>\n      <th>2018-02-14</th>\n      <td>21.49</td>\n      <td>21.92</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>2015-03-06</th>\n      <td>13.17</td>\n      <td>14.28</td>\n    </tr>\n    <tr>\n      <th>2015-03-05</th>\n      <td>12.88</td>\n      <td>13.16</td>\n    </tr>\n    <tr>\n      <th>2015-03-04</th>\n      <td>12.80</td>\n      <td>12.90</td>\n    </tr>\n    <tr>\n      <th>2015-03-03</th>\n      <td>12.52</td>\n      <td>12.70</td>\n    </tr>\n    <tr>\n      <th>2015-03-02</th>\n      <td>12.25</td>\n      <td>12.52</td>\n    </tr>\n  </tbody>\n</table>\n<p>643 rows × 2 columns</p>\n</div>"
     },
     "execution_count": 15,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "import pandas as pd\n",
    "# 读取文件,并且指定只获取'open', 'close'指标\n",
    "data = pd.read_csv(\"stock_day.csv\", usecols=['open', 'close'])\n",
    "data"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:39:29.056807200Z",
     "start_time": "2024-02-22T07:39:29.049550100Z"
    }
   },
   "id": "70150511a276df50",
   "execution_count": 15
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 1.2 to_csv\n",
    "DataFrame.to_csv(path_or_buf=None, sep=', ’, columns=None, header=True, index=True, mode='w', encoding=None)\n",
    "参数：\n",
    "- path_or_buf :文件路径\n",
    "- sep :分隔符，默认用\",\"隔开\n",
    "- columns :选择需要的列索引\n",
    "- header :boolean or list of string, default True,是否写进列索引值\n",
    "- index:是否写进行索引\n",
    "- mode:'w'：重写, 'a' 追加\n",
    "\n",
    "举例：保存读取出来的股票数据\n",
    "- 保存'open'列的数据，然后读取查看结果"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "59c410e32fbf354a"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "   Unnamed: 0   open\n0  2018-02-27  23.53\n1  2018-02-26  22.80\n2  2018-02-23  22.88\n3  2018-02-22  22.25\n4  2018-02-14  21.49\n5  2018-02-13  21.40\n6  2018-02-12  20.70\n7  2018-02-09  21.20\n8  2018-02-08  21.79\n9  2018-02-07  22.69",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Unnamed: 0</th>\n      <th>open</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>2018-02-27</td>\n      <td>23.53</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2018-02-26</td>\n      <td>22.80</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>2018-02-23</td>\n      <td>22.88</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>2018-02-22</td>\n      <td>22.25</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>2018-02-14</td>\n      <td>21.49</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>2018-02-13</td>\n      <td>21.40</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>2018-02-12</td>\n      <td>20.70</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>2018-02-09</td>\n      <td>21.20</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>2018-02-08</td>\n      <td>21.79</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>2018-02-07</td>\n      <td>22.69</td>\n    </tr>\n  </tbody>\n</table>\n</div>"
     },
     "execution_count": 18,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 选取10行数据保存,便于观察数据\n",
    "data[:10].to_csv(\"test.csv\", columns=['open'])\n",
    "# 读取，查看结果\n",
    "pd.read_csv(\"test.csv\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:39:55.467533900Z",
     "start_time": "2024-02-22T07:39:55.457495900Z"
    }
   },
   "id": "65ce8d44c1979047",
   "execution_count": 18
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "# index:存储不会将索引值变成一列数据\n",
    "data[:10].to_csv(\"test.csv\", columns=['open'], index=False)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:40:37.761669600Z",
     "start_time": "2024-02-22T07:40:37.752850800Z"
    }
   },
   "id": "84d47cc40bb6452d",
   "execution_count": 19
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 2 HDF5\n",
    "## 2.1 read_hdf与to_hdf\n",
    "HDF5文件的读取和存储需要指定一个键，值为要存储的DataFrame\n",
    "pandas.read_hdf(path_or_buf，key =None，** kwargs)\n",
    "# 从h5文件当中读取数据\n",
    "参数：\n",
    "- path_or_buffer:文件路径\n",
    "- key:读取的键\n",
    "- return:Theselected object\n",
    "> DataFrame.to_hdf(path_or_buf, *key*, **\\*kwargs*)\n",
    "## 2.2 案例\n",
    "- 读取文件"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "d715e95fd004d44"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "      000001.SZ  000002.SZ  000004.SZ  000005.SZ  000006.SZ  000007.SZ  \\\n0         16.30      17.71       4.58       2.88      14.60       2.62   \n1         17.02      19.20       4.65       3.02      15.97       2.65   \n2         17.02      17.28       4.56       3.06      14.37       2.63   \n3         16.18      16.97       4.49       2.95      13.10       2.73   \n4         16.95      17.19       4.55       2.99      13.18       2.77   \n...         ...        ...        ...        ...        ...        ...   \n2673      12.96      35.99      22.84       4.37       9.85      16.66   \n2674      13.08      35.84      23.02       4.41       9.85      16.66   \n2675      13.47      35.67      22.40       4.32       9.85      16.66   \n2676      13.40      35.15      22.29       4.29       9.85      16.66   \n2677      13.55      35.55      22.20       4.37       9.85      16.66   \n\n      000008.SZ  000009.SZ  000010.SZ  000011.SZ  ...  001965.SZ  603283.SH  \\\n0          4.96       4.66       5.37       6.02  ...        NaN        NaN   \n1          4.95       4.70       5.37       6.27  ...        NaN        NaN   \n2          4.82       4.47       5.37       5.96  ...        NaN        NaN   \n3          4.89       4.33       5.37       5.77  ...        NaN        NaN   \n4          4.97       4.42       5.37       5.92  ...        NaN        NaN   \n...         ...        ...        ...        ...  ...        ...        ...   \n2673       8.47       7.52       6.20      17.88  ...      12.99      23.42   \n2674       8.49       7.48       6.01      17.75  ...      12.83      25.76   \n2675       8.49       7.38       5.97      17.45  ...      12.20      28.34   \n2676       8.56       7.04       5.84      17.49  ...      12.11      31.17   \n2677       8.67       7.06       5.99      17.76  ...      11.91      34.29   \n\n      002920.SZ  002921.SZ  300684.SZ  002922.SZ  300735.SZ  603329.SH  \\\n0           NaN        NaN        NaN        NaN        NaN        NaN   \n1           NaN        NaN        NaN        NaN        NaN        NaN   \n2           NaN        NaN        NaN        NaN        NaN        NaN   \n3           NaN        NaN        NaN        NaN        NaN        NaN   \n4           NaN        NaN        NaN        NaN        NaN        NaN   \n...         ...        ...        ...        ...        ...        ...   \n2673      47.99      32.40      22.45      28.79      23.18      24.45   \n2674      45.14      35.64      24.70      31.67      25.50      26.90   \n2675      43.21      39.20      27.17      34.84      28.05      29.59   \n2676      43.76      40.88      29.89      34.84      29.64      32.55   \n2677      41.71      39.10      32.88      34.84      27.92      31.82   \n\n      603655.SH  603080.SH  \n0           NaN        NaN  \n1           NaN        NaN  \n2           NaN        NaN  \n3           NaN        NaN  \n4           NaN        NaN  \n...         ...        ...  \n2673      14.98      26.06  \n2674      16.48      28.67  \n2675      18.13      31.54  \n2676      19.94      34.69  \n2677      21.93      38.16  \n\n[2678 rows x 3562 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>000001.SZ</th>\n      <th>000002.SZ</th>\n      <th>000004.SZ</th>\n      <th>000005.SZ</th>\n      <th>000006.SZ</th>\n      <th>000007.SZ</th>\n      <th>000008.SZ</th>\n      <th>000009.SZ</th>\n      <th>000010.SZ</th>\n      <th>000011.SZ</th>\n      <th>...</th>\n      <th>001965.SZ</th>\n      <th>603283.SH</th>\n      <th>002920.SZ</th>\n      <th>002921.SZ</th>\n      <th>300684.SZ</th>\n      <th>002922.SZ</th>\n      <th>300735.SZ</th>\n      <th>603329.SH</th>\n      <th>603655.SH</th>\n      <th>603080.SH</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>16.30</td>\n      <td>17.71</td>\n      <td>4.58</td>\n      <td>2.88</td>\n      <td>14.60</td>\n      <td>2.62</td>\n      <td>4.96</td>\n      <td>4.66</td>\n      <td>5.37</td>\n      <td>6.02</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>17.02</td>\n      <td>19.20</td>\n      <td>4.65</td>\n      <td>3.02</td>\n      <td>15.97</td>\n      <td>2.65</td>\n      <td>4.95</td>\n      <td>4.70</td>\n      <td>5.37</td>\n      <td>6.27</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>17.02</td>\n      <td>17.28</td>\n      <td>4.56</td>\n      <td>3.06</td>\n      <td>14.37</td>\n      <td>2.63</td>\n      <td>4.82</td>\n      <td>4.47</td>\n      <td>5.37</td>\n      <td>5.96</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>16.18</td>\n      <td>16.97</td>\n      <td>4.49</td>\n      <td>2.95</td>\n      <td>13.10</td>\n      <td>2.73</td>\n      <td>4.89</td>\n      <td>4.33</td>\n      <td>5.37</td>\n      <td>5.77</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>16.95</td>\n      <td>17.19</td>\n      <td>4.55</td>\n      <td>2.99</td>\n      <td>13.18</td>\n      <td>2.77</td>\n      <td>4.97</td>\n      <td>4.42</td>\n      <td>5.37</td>\n      <td>5.92</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>2673</th>\n      <td>12.96</td>\n      <td>35.99</td>\n      <td>22.84</td>\n      <td>4.37</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.47</td>\n      <td>7.52</td>\n      <td>6.20</td>\n      <td>17.88</td>\n      <td>...</td>\n      <td>12.99</td>\n      <td>23.42</td>\n      <td>47.99</td>\n      <td>32.40</td>\n      <td>22.45</td>\n      <td>28.79</td>\n      <td>23.18</td>\n      <td>24.45</td>\n      <td>14.98</td>\n      <td>26.06</td>\n    </tr>\n    <tr>\n      <th>2674</th>\n      <td>13.08</td>\n      <td>35.84</td>\n      <td>23.02</td>\n      <td>4.41</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.49</td>\n      <td>7.48</td>\n      <td>6.01</td>\n      <td>17.75</td>\n      <td>...</td>\n      <td>12.83</td>\n      <td>25.76</td>\n      <td>45.14</td>\n      <td>35.64</td>\n      <td>24.70</td>\n      <td>31.67</td>\n      <td>25.50</td>\n      <td>26.90</td>\n      <td>16.48</td>\n      <td>28.67</td>\n    </tr>\n    <tr>\n      <th>2675</th>\n      <td>13.47</td>\n      <td>35.67</td>\n      <td>22.40</td>\n      <td>4.32</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.49</td>\n      <td>7.38</td>\n      <td>5.97</td>\n      <td>17.45</td>\n      <td>...</td>\n      <td>12.20</td>\n      <td>28.34</td>\n      <td>43.21</td>\n      <td>39.20</td>\n      <td>27.17</td>\n      <td>34.84</td>\n      <td>28.05</td>\n      <td>29.59</td>\n      <td>18.13</td>\n      <td>31.54</td>\n    </tr>\n    <tr>\n      <th>2676</th>\n      <td>13.40</td>\n      <td>35.15</td>\n      <td>22.29</td>\n      <td>4.29</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.56</td>\n      <td>7.04</td>\n      <td>5.84</td>\n      <td>17.49</td>\n      <td>...</td>\n      <td>12.11</td>\n      <td>31.17</td>\n      <td>43.76</td>\n      <td>40.88</td>\n      <td>29.89</td>\n      <td>34.84</td>\n      <td>29.64</td>\n      <td>32.55</td>\n      <td>19.94</td>\n      <td>34.69</td>\n    </tr>\n    <tr>\n      <th>2677</th>\n      <td>13.55</td>\n      <td>35.55</td>\n      <td>22.20</td>\n      <td>4.37</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.67</td>\n      <td>7.06</td>\n      <td>5.99</td>\n      <td>17.76</td>\n      <td>...</td>\n      <td>11.91</td>\n      <td>34.29</td>\n      <td>41.71</td>\n      <td>39.10</td>\n      <td>32.88</td>\n      <td>34.84</td>\n      <td>27.92</td>\n      <td>31.82</td>\n      <td>21.93</td>\n      <td>38.16</td>\n    </tr>\n  </tbody>\n</table>\n<p>2678 rows × 3562 columns</p>\n</div>"
     },
     "execution_count": 22,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "day_close = pd.read_hdf(\"day_close.h5\") #报错则 pip install tables\n",
    "day_close"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:45:38.047672800Z",
     "start_time": "2024-02-22T07:45:37.865832500Z"
    }
   },
   "id": "f993d191398d2d2",
   "execution_count": 22
  },
  {
   "cell_type": "markdown",
   "source": [
    "- 存储文件"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "3ca48b255d5faad"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "day_close.to_hdf(\"test.h5\", key=\"day_close\")"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:46:32.563539400Z",
     "start_time": "2024-02-22T07:46:32.363602400Z"
    }
   },
   "id": "4777096bfae3dffe",
   "execution_count": 23
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "      000001.SZ  000002.SZ  000004.SZ  000005.SZ  000006.SZ  000007.SZ  \\\n0         16.30      17.71       4.58       2.88      14.60       2.62   \n1         17.02      19.20       4.65       3.02      15.97       2.65   \n2         17.02      17.28       4.56       3.06      14.37       2.63   \n3         16.18      16.97       4.49       2.95      13.10       2.73   \n4         16.95      17.19       4.55       2.99      13.18       2.77   \n...         ...        ...        ...        ...        ...        ...   \n2673      12.96      35.99      22.84       4.37       9.85      16.66   \n2674      13.08      35.84      23.02       4.41       9.85      16.66   \n2675      13.47      35.67      22.40       4.32       9.85      16.66   \n2676      13.40      35.15      22.29       4.29       9.85      16.66   \n2677      13.55      35.55      22.20       4.37       9.85      16.66   \n\n      000008.SZ  000009.SZ  000010.SZ  000011.SZ  ...  001965.SZ  603283.SH  \\\n0          4.96       4.66       5.37       6.02  ...        NaN        NaN   \n1          4.95       4.70       5.37       6.27  ...        NaN        NaN   \n2          4.82       4.47       5.37       5.96  ...        NaN        NaN   \n3          4.89       4.33       5.37       5.77  ...        NaN        NaN   \n4          4.97       4.42       5.37       5.92  ...        NaN        NaN   \n...         ...        ...        ...        ...  ...        ...        ...   \n2673       8.47       7.52       6.20      17.88  ...      12.99      23.42   \n2674       8.49       7.48       6.01      17.75  ...      12.83      25.76   \n2675       8.49       7.38       5.97      17.45  ...      12.20      28.34   \n2676       8.56       7.04       5.84      17.49  ...      12.11      31.17   \n2677       8.67       7.06       5.99      17.76  ...      11.91      34.29   \n\n      002920.SZ  002921.SZ  300684.SZ  002922.SZ  300735.SZ  603329.SH  \\\n0           NaN        NaN        NaN        NaN        NaN        NaN   \n1           NaN        NaN        NaN        NaN        NaN        NaN   \n2           NaN        NaN        NaN        NaN        NaN        NaN   \n3           NaN        NaN        NaN        NaN        NaN        NaN   \n4           NaN        NaN        NaN        NaN        NaN        NaN   \n...         ...        ...        ...        ...        ...        ...   \n2673      47.99      32.40      22.45      28.79      23.18      24.45   \n2674      45.14      35.64      24.70      31.67      25.50      26.90   \n2675      43.21      39.20      27.17      34.84      28.05      29.59   \n2676      43.76      40.88      29.89      34.84      29.64      32.55   \n2677      41.71      39.10      32.88      34.84      27.92      31.82   \n\n      603655.SH  603080.SH  \n0           NaN        NaN  \n1           NaN        NaN  \n2           NaN        NaN  \n3           NaN        NaN  \n4           NaN        NaN  \n...         ...        ...  \n2673      14.98      26.06  \n2674      16.48      28.67  \n2675      18.13      31.54  \n2676      19.94      34.69  \n2677      21.93      38.16  \n\n[2678 rows x 3562 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>000001.SZ</th>\n      <th>000002.SZ</th>\n      <th>000004.SZ</th>\n      <th>000005.SZ</th>\n      <th>000006.SZ</th>\n      <th>000007.SZ</th>\n      <th>000008.SZ</th>\n      <th>000009.SZ</th>\n      <th>000010.SZ</th>\n      <th>000011.SZ</th>\n      <th>...</th>\n      <th>001965.SZ</th>\n      <th>603283.SH</th>\n      <th>002920.SZ</th>\n      <th>002921.SZ</th>\n      <th>300684.SZ</th>\n      <th>002922.SZ</th>\n      <th>300735.SZ</th>\n      <th>603329.SH</th>\n      <th>603655.SH</th>\n      <th>603080.SH</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>16.30</td>\n      <td>17.71</td>\n      <td>4.58</td>\n      <td>2.88</td>\n      <td>14.60</td>\n      <td>2.62</td>\n      <td>4.96</td>\n      <td>4.66</td>\n      <td>5.37</td>\n      <td>6.02</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>17.02</td>\n      <td>19.20</td>\n      <td>4.65</td>\n      <td>3.02</td>\n      <td>15.97</td>\n      <td>2.65</td>\n      <td>4.95</td>\n      <td>4.70</td>\n      <td>5.37</td>\n      <td>6.27</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>17.02</td>\n      <td>17.28</td>\n      <td>4.56</td>\n      <td>3.06</td>\n      <td>14.37</td>\n      <td>2.63</td>\n      <td>4.82</td>\n      <td>4.47</td>\n      <td>5.37</td>\n      <td>5.96</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>16.18</td>\n      <td>16.97</td>\n      <td>4.49</td>\n      <td>2.95</td>\n      <td>13.10</td>\n      <td>2.73</td>\n      <td>4.89</td>\n      <td>4.33</td>\n      <td>5.37</td>\n      <td>5.77</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>16.95</td>\n      <td>17.19</td>\n      <td>4.55</td>\n      <td>2.99</td>\n      <td>13.18</td>\n      <td>2.77</td>\n      <td>4.97</td>\n      <td>4.42</td>\n      <td>5.37</td>\n      <td>5.92</td>\n      <td>...</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>2673</th>\n      <td>12.96</td>\n      <td>35.99</td>\n      <td>22.84</td>\n      <td>4.37</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.47</td>\n      <td>7.52</td>\n      <td>6.20</td>\n      <td>17.88</td>\n      <td>...</td>\n      <td>12.99</td>\n      <td>23.42</td>\n      <td>47.99</td>\n      <td>32.40</td>\n      <td>22.45</td>\n      <td>28.79</td>\n      <td>23.18</td>\n      <td>24.45</td>\n      <td>14.98</td>\n      <td>26.06</td>\n    </tr>\n    <tr>\n      <th>2674</th>\n      <td>13.08</td>\n      <td>35.84</td>\n      <td>23.02</td>\n      <td>4.41</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.49</td>\n      <td>7.48</td>\n      <td>6.01</td>\n      <td>17.75</td>\n      <td>...</td>\n      <td>12.83</td>\n      <td>25.76</td>\n      <td>45.14</td>\n      <td>35.64</td>\n      <td>24.70</td>\n      <td>31.67</td>\n      <td>25.50</td>\n      <td>26.90</td>\n      <td>16.48</td>\n      <td>28.67</td>\n    </tr>\n    <tr>\n      <th>2675</th>\n      <td>13.47</td>\n      <td>35.67</td>\n      <td>22.40</td>\n      <td>4.32</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.49</td>\n      <td>7.38</td>\n      <td>5.97</td>\n      <td>17.45</td>\n      <td>...</td>\n      <td>12.20</td>\n      <td>28.34</td>\n      <td>43.21</td>\n      <td>39.20</td>\n      <td>27.17</td>\n      <td>34.84</td>\n      <td>28.05</td>\n      <td>29.59</td>\n      <td>18.13</td>\n      <td>31.54</td>\n    </tr>\n    <tr>\n      <th>2676</th>\n      <td>13.40</td>\n      <td>35.15</td>\n      <td>22.29</td>\n      <td>4.29</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.56</td>\n      <td>7.04</td>\n      <td>5.84</td>\n      <td>17.49</td>\n      <td>...</td>\n      <td>12.11</td>\n      <td>31.17</td>\n      <td>43.76</td>\n      <td>40.88</td>\n      <td>29.89</td>\n      <td>34.84</td>\n      <td>29.64</td>\n      <td>32.55</td>\n      <td>19.94</td>\n      <td>34.69</td>\n    </tr>\n    <tr>\n      <th>2677</th>\n      <td>13.55</td>\n      <td>35.55</td>\n      <td>22.20</td>\n      <td>4.37</td>\n      <td>9.85</td>\n      <td>16.66</td>\n      <td>8.67</td>\n      <td>7.06</td>\n      <td>5.99</td>\n      <td>17.76</td>\n      <td>...</td>\n      <td>11.91</td>\n      <td>34.29</td>\n      <td>41.71</td>\n      <td>39.10</td>\n      <td>32.88</td>\n      <td>34.84</td>\n      <td>27.92</td>\n      <td>31.82</td>\n      <td>21.93</td>\n      <td>38.16</td>\n    </tr>\n  </tbody>\n</table>\n<p>2678 rows × 3562 columns</p>\n</div>"
     },
     "execution_count": 26,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 再次读取的时候, 需要指定键的名字\n",
    "new_close = pd.read_hdf(\"test.h5\", key=\"day_close\")\n",
    "new_close"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:47:33.458950800Z",
     "start_time": "2024-02-22T07:47:33.338617100Z"
    }
   },
   "id": "977cea03cc7646c0",
   "execution_count": 26
  },
  {
   "cell_type": "markdown",
   "source": [
    "注意：优先选择使用HDF5文件存储\n",
    "- HDF5在存储的时候支持压缩，使用的方式是blosc，这个是速度最快的也是pandas默认支持的使用压缩可以提磁盘利用率，节省空间\n",
    "- HDF5还是跨平台的，可以轻松迁移到hadoop 上面注意：优先选择使用HDF5文件存储"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "4f4ca05a4dd74c2e"
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 3 JSON\n",
    "## 3.1 read_json\n",
    "pandas.read_json(path_or_buf=None, orient=None, typ='frame', lines=False)\n",
    "# 将JSON格式准换成默认的Pandas DataFrame格式\n",
    "参数：\n",
    "- orient : string,Indication of expected JSON string format.\n",
    "    - 'split' : dict like {index -> [index], columns -> [columns], data -> [values]}\n",
    "    - split 将索引总结到索引，列名到列名，数据到数据。将三部分都分开了\n",
    "    - 'records' : list like [{column -> value}, ... , {column -> value}]\n",
    "    - records 以columns：values的形式输出\n",
    "    - 'index' : dict like {index -> {column -> value}}\n",
    "    - index 以index：{columns：values}...的形式输出\n",
    "    - 'columns' : dict like {column -> {index -> value}},默认该格式\n",
    "    - colums 以columns:{index:values}的形式输出\n",
    "    - 'values' : just the values array\n",
    "    - values 直接输出值\n",
    "- lines : boolean, default False\n",
    "    - 按照每行读取json对象\n",
    "- typ : default ‘frame’， 指定转换成的对象类型series或者dataframe\n",
    "\n",
    "read_josn 案例：\n",
    "- 数据介绍\n",
    "这里使用一个新闻标题讽刺数据集，格式为json。is_sarcastic：1讽刺的，否则为0；headline：新闻报道的标题；article_link：链接到原始新闻文章。存储格式为：\n",
    "\n",
    "```json\n",
    "{\"article_link\": \"https://www.huffingtonpost.com/entry/versace-black-code_us_5861fbefe4b0de3a08f600d5\", \"headline\": \"former versace store clerk sues over secret 'black code' for minority shoppers\", \"is_sarcastic\": 0}\n",
    "{\"article_link\": \"https://www.huffingtonpost.com/entry/roseanne-revival-review_us_5ab3a497e4b054d118e04365\", \"headline\": \"the 'roseanne' revival catches up to our thorny political mood, for better and worse\", \"is_sarcastic\": 0}\n",
    "```\n",
    "\n",
    "- 读取\n",
    "orient指定存储的json格式，lines指定按照行去变成一个样本"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "8e681e735fc31e8f"
  },
  {
   "cell_type": "code",
   "outputs": [
    {
     "data": {
      "text/plain": "                                            article_link  \\\n0      https://www.huffingtonpost.com/entry/versace-b...   \n1      https://www.huffingtonpost.com/entry/roseanne-...   \n2      https://local.theonion.com/mom-starting-to-fea...   \n3      https://politics.theonion.com/boehner-just-wan...   \n4      https://www.huffingtonpost.com/entry/jk-rowlin...   \n...                                                  ...   \n26704  https://www.huffingtonpost.com/entry/american-...   \n26705  https://www.huffingtonpost.com/entry/americas-...   \n26706  https://www.huffingtonpost.com/entry/reparatio...   \n26707  https://www.huffingtonpost.com/entry/israeli-b...   \n26708  https://www.huffingtonpost.com/entry/gourmet-g...   \n\n                                                headline  is_sarcastic  \n0      former versace store clerk sues over secret 'b...             0  \n1      the 'roseanne' revival catches up to our thorn...             0  \n2      mom starting to fear son's web series closest ...             1  \n3      boehner just wants wife to listen, not come up...             1  \n4      j.k. rowling wishes snape happy birthday in th...             0  \n...                                                  ...           ...  \n26704               american politics in moral free-fall             0  \n26705                            america's best 20 hikes             0  \n26706                              reparations and obama             0  \n26707  israeli ban targeting boycott supporters raise...             0  \n26708                  gourmet gifts for the foodie 2014             0  \n\n[26709 rows x 3 columns]",
      "text/html": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>article_link</th>\n      <th>headline</th>\n      <th>is_sarcastic</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>https://www.huffingtonpost.com/entry/versace-b...</td>\n      <td>former versace store clerk sues over secret 'b...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>https://www.huffingtonpost.com/entry/roseanne-...</td>\n      <td>the 'roseanne' revival catches up to our thorn...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>https://local.theonion.com/mom-starting-to-fea...</td>\n      <td>mom starting to fear son's web series closest ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>https://politics.theonion.com/boehner-just-wan...</td>\n      <td>boehner just wants wife to listen, not come up...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>https://www.huffingtonpost.com/entry/jk-rowlin...</td>\n      <td>j.k. rowling wishes snape happy birthday in th...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>...</th>\n      <td>...</td>\n      <td>...</td>\n      <td>...</td>\n    </tr>\n    <tr>\n      <th>26704</th>\n      <td>https://www.huffingtonpost.com/entry/american-...</td>\n      <td>american politics in moral free-fall</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>26705</th>\n      <td>https://www.huffingtonpost.com/entry/americas-...</td>\n      <td>america's best 20 hikes</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>26706</th>\n      <td>https://www.huffingtonpost.com/entry/reparatio...</td>\n      <td>reparations and obama</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>26707</th>\n      <td>https://www.huffingtonpost.com/entry/israeli-b...</td>\n      <td>israeli ban targeting boycott supporters raise...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>26708</th>\n      <td>https://www.huffingtonpost.com/entry/gourmet-g...</td>\n      <td>gourmet gifts for the foodie 2014</td>\n      <td>0</td>\n    </tr>\n  </tbody>\n</table>\n<p>26709 rows × 3 columns</p>\n</div>"
     },
     "execution_count": 29,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "json_read = pd.read_json(\"Sarcasm_Headlines_Dataset.json\", orient=\"records\", lines=True)\n",
    "json_read"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:51:56.251878300Z",
     "start_time": "2024-02-22T07:51:56.191748600Z"
    }
   },
   "id": "fc38376cc2c7f76",
   "execution_count": 29
  },
  {
   "cell_type": "markdown",
   "source": [
    "## 3.2 to_json\n",
    "DataFrame.to_json(*path_or_buf=None*, *orient=None*, *lines=False*)\n",
    "参数：\n",
    "- Pandas 对象存储为json格式\n",
    "- ath_or_buf=None：文件地址\n",
    "- rient:存储的json形式，{‘split’,’records’,’index’,’columns’,’values’}\n",
    "- ines:一个对象存储为一行\n",
    "\n",
    "案例：\n",
    "- 存储文件"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "aebcc5130cb8c40e"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "json_read.to_json(\"test.json\", orient='records')"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:52:41.775147100Z",
     "start_time": "2024-02-22T07:52:41.747404200Z"
    }
   },
   "id": "f5f0c745bed23d9c",
   "execution_count": 30
  },
  {
   "cell_type": "markdown",
   "source": [
    "- 修改lines参数为True"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "5271200bb6108bc3"
  },
  {
   "cell_type": "code",
   "outputs": [],
   "source": [
    "json_read.to_json(\"test.json\", orient='records', lines=True)"
   ],
   "metadata": {
    "collapsed": false,
    "ExecuteTime": {
     "end_time": "2024-02-22T07:53:03.797159200Z",
     "start_time": "2024-02-22T07:53:03.775561900Z"
    }
   },
   "id": "ebda9c72f34a5d6d",
   "execution_count": 31
  },
  {
   "cell_type": "markdown",
   "source": [
    "# 4 小结\n",
    "- pandas的CSV、HDF5、JSON文件的读取【知道】\n",
    "- 对象.read_**()\n",
    "- 对象.to_**()"
   ],
   "metadata": {
    "collapsed": false
   },
   "id": "71731011622dfe50"
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 2
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython2",
   "version": "2.7.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
